Papers with Video Question Answering
AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Currently, personal AI assistants on the phone and AR glasses can assist our daily life in addressing our questions like "how to adjust the date for this watch?" |
| Approach: | They propose a task that asks a question about affordance of items in our daily life . they construct a dataset that contains 3.2k multimodal questions on 1.6k video segments . |
| Outcome: | The proposed task outperforms baseline methods while still having room for improvement in the future. |
ReasVQA: Advancing VideoQA with Imperfect Reasoning Process (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to VideoQA often fail when complex reasoning or temporal relationships are involved. |
| Approach: | They propose a method that leverages reasoning processes generated by Multimodal Large Language Models to improve VideoQA models. |
| Outcome: | The proposed method improves VideoQA models on three benchmarks. |
Multi-Scale Progressive Attention Network for Video Question Answering (2021.acl-short)
Copied to clipboard
| Challenge: | Experimental evaluations on three benchmarks: TGIF-QA, MSVD-QA and MSRVTT-QA show our method has achieved state-of-the-art performance. |
| Approach: | They propose a multi-scale progressive attention network to fuse visual and text information. |
| Outcome: | The proposed method achieves state-of-the-art on three benchmarks: TGIF-QA, MSVD-QA and MSRVTT-QA. |
HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020.emnlp-main)
Copied to clipboard
| Challenge: | HERO is a framework for large-scale video+language omni-representation learning. |
| Approach: | They propose a framework for large-scale video+language omni-representation learning that encodes multimodal inputs in a hierarchical structure and uses Masked Language Modeling and Masked Frame Modeling to train models. |
| Outcome: | The proposed framework achieves state-of-the-art on multiple benchmarks over text-based video/video-moment retrieval, video question answering (QA), Video-and-language Inference and video Captioning tasks across different domains. |
Exploring Question Guidance and Answer Calibration for Visually Grounded Video Question Answering (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for videoQA lack temporal localization labels, leading to inaccurate localization. |
| Approach: | They propose a Question-Guided and Answer-Calibrated TRansformer which guides and calibrates localization using question and option texts without localization labels. |
| Outcome: | The proposed model achieves comparable accuracy to large-scale pretrained models and leads in localization aspects. |
Video Question Answering with Phrases via Semantic Roles (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing VidQA evaluation metrics limit the models’ application scenario to a single-word answer or selecting a phrase from a fixed set of phrases. |
| Approach: | They propose to leverage video descriptions to mask out certain phrases to enable evaluation of answer phrases. |
| Outcome: | The proposed model reduces the influence of language bias on VidQA datasets by retrieving a video having a different answer for the same question. |
Large Language Models are Temporal and Causal Reasoners for Video Question Answering (2023.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable performances on a wide range of natural language understanding and generation tasks. |
| Approach: | They propose a framework that exploits linguistic shortcuts and mitigates 'linguistic bias' by flipping the source pair and target label to understand their complex relationships. |
| Outcome: | The proposed framework outperforms both LLMs-based and non-LLMs- based models on five challenging VideoQA benchmarks. |
Mulan: A Multi-Level Alignment Model for Video Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods focus on visual-language alignment at the video level, but they do not account for fine-grained semantic interaction between video and text. |
| Approach: | They propose a multi-level Alignment Model for Video Question Answering that establishes alignment between visual and textual modalities at the object-level, frame-level and video-level. |
| Outcome: | The proposed model outperforms state-of-the-art methods even with a small amount of extra visual-language pre-training data and a reduced number of trainable parameters. |
Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering (2021.acl-long)
Copied to clipboard
| Challenge: | Recent advances in natural language processing and computer vision have made significant progress in artificial intelligence (AI). |
| Approach: | They propose Motion-Appearance Synergistic Networks which embed cross-modal features grounded on motion and appearance information and selectively utilize them depending on the question’s intentions. |
| Outcome: | The proposed network achieves state-of-the-art on the TGIF-QA and MSVD-QA datasets and qualitatively analyzes the results. |
Rethinking Multi-Modal Alignment in Multi-Choice VideoQA from Feature and Sample Perspectives (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to VideoQA focus on utilizing frame- or object-level visual representations, but they neglect visual-language interactions. |
| Approach: | They propose to break down video into trajectories and first leverage trajectory feature in VideoQA to enhance alignment between two modalities. |
| Outcome: | The proposed method outperforms all the state-of-the-art models on the NExT-QA benchmark. |
GazeVQA: A Video Question Answering Dataset for Multiview Eye-Gaze Task-Oriented Collaborations (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on the use of exocentric and egocentric videos in video question answering are focusing on eye-gaze information. |
| Approach: | They propose a task-oriented VQA dataset that captures eye-gaze information . they propose assisting models that ground the perceptual input into semantic information based on three different answer types . |
| Outcome: | The proposed model can ground the perceptual input into semantic information while reducing ambiguities. |
ViLL-E: Video LLM Embeddings for Retrieval (2026.acl-long)
Copied to clipboard
| Challenge: | Video Large Language Models excel at video understanding tasks where outputs are textual . however, they underperform specialized embedding-based models in Retrieval tasks . |
| Approach: | They propose a video-LLM-based model with an embedding generation mechanism that allows the model to "think longer" for complex videos and stop early for easy ones. |
| Outcome: | The proposed model outperforms specialized embedding-based models in video understanding tasks while remaining competitive on VideoQA tasks. |